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EC  PERIETAL  STUDY  OF  THE 
NATURAL  PILOT  FLIGHT  PROFICIENCY  EVALUATION  MODEL 


ABSTRACT 


This  experiment  investigated  two  major  hypotheses  generated  by  the 
Natural  Pilot  Model  for  Flight  Proficiency  Evaluation.  These  specify  that 
human  adaptability  and  economy  of  effort  as  understood  in  the  context  of 
servo-mechanism  theory  are  important  parameters  of  flying  proficiency,  and 
serve  to  differentiate  proficient  ("natural")  from  poor  ("mechanical") 
operators.  Adaptability  was  studied  by  systematically  varying  the  controlled- 
element  dynamics  (control-display  relationships);  economy  of  effort,  by 
varying  the  percent  of  time  during  which  information  was  displayed  (target 
intermittency).  For  all  experimental  conditions,  the  proficient  trackers 
retained  superiority  over  the  poor  ones.  For  intermittency  conditions, 
performance  decrement  was  the  same  for  proficient  and  poor  trackers.  For 
the  condition  of  changing  control-display  dynamics,  one  of  the  three  measures 
of  system  performance  showed  less  variation  and  less  decrement  for  the 
proficient  than  for  the  poor  trackers.  Conventional  tracking  practice  did 
not  Improve  performance  as  measured  by  the  adaptation  and  economy  of  effort 
criteria. 


This  study  lends  support  to  the  basic  hypotheses  and  therefore  raises 
hope  that  this  approach,  which  would  obviate  many  of  the  criticisms  of 
conventional  proficiency  measurement,  is  sound.  However  it  also  reveals 
difficulties  in  obtaining  a  cogent  test  of  a  model  that  attempts  to  combine 
servo-mechanism,  perceptual -motor  learning,  and  psychometric  concepts. 
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FOREWORD 


The  measurement  of  pilot  proficiency  either  in  the  OFT  or  the  actual 
aircraft  has  been  beset  by  a  multitude  of  theoretical  and  practical 
difficulties.  This  is  due  to  mary  factors,  but  especially  to  rater  un¬ 
reliability,  task  variations,  differences  between  simulated  and  real 
flight  situations,  and  gaps  in  the  understanding  of  the  basic  nature  of 
the  flying  task. 

A  new  approach  was  taken  in  our  studies  at  the  Franklin  Institute. 

Instead  of  specifying  and  describing  the  correct  ways  of  performing  a 
representative  series  of  maneuvers,  and  then  determining  appropriate 
tolerances,  the  research  aimed  at  evolving  a  generalized  method  that  could 
be  applied  to  any  maneuver,  and  that  would  be  valid  regardless  of  air¬ 
craft  type  or  the  level  of  experience  of  the  pilotj  and  regardless  of 
whether  measurement  took  place  in  the  simulator  or  the  aircraft. 

During  Phase  I  of  this  program  (TR  NAVT-IADEVCS!  323-1) »  the  ability  of 
test  pilots  to  evaluate  the  handling  qualities  of  new  aircraft,  and  the 
ability  of  LSO's  (landing  signal  officers)  to  judge  the  correctness  of 
pilot-aircraft  performance  were  interpreted  from  the  servo-mechanism 
viewpoint  of  skill-development.  This  led  to  the  "natural  pilot"  model 
which  hypothesized  that  the  most  efficient  way  to  evaluate  flight-control 
skills  would  be  in  terms  of  the  pilot-aircraft  system's  consistency  of 
performance  despite  vicissitudes  of  flight  or  mission;  and  in  terms  of 
the  economy  of  effort  which  characterized  the  pilot's  performance. 

The  first  of  these  variables  or  factors  is  related  to  the  pilot's 
adaptability,  which  is  considered  the  human's  most  essential  and  unique 
contribution  to  the  man-machine  system;  the  second  rests  on  the  fact  that 
highly-skilled  performance  is  oharao  rized  by  conservation  or  economy 
of  effort  ' "least  effort")  and  the  n.u  .  for  less  redundancy  of  information 
to  accomplish  its  goal. 

Phase  II,  the  present  stuty,  constitutes  an  experimental  test  of  both 
hypotheses.  A  compensatory-type  tracking  apparatus  was  used  to  obtain 
normative  data  of  the  stability  of  human  performance  during  changes  in 
control-displ^  dynamics,  and  during  forced  economy  of  effort.  The 
latter  condition  was  elicited  by  systematically  curtailing  the  percent 
of  time  that  information  was  displayed  to  the  operator  while  tracking. 

The  data  showed  a  considerable  relationship  between  generalized  tracking 
ability  and  each  of  the  two  hypothesized  factors  of  adaptability  and 
econony  of  effort.  The  data  also  showed  that  the  two  hypothesized  factors 
are  independent;  and  that  ordinary  training,  i.e.,  training  not  predicated 
on  the  special  need  for  adaptability  and  economy  of  effort,  was  not 
successful  in  improving  performance  on  these  factors. 

The  findings  substantiate  the  hypotheses.  However,  because  of  the 
difficulties  encountered  in  testing  our  hypotheses,  and  in  merging  servo¬ 
mechanism,  psycho-motor,  and  tests-and-measurement  concepts— more  research, 
especially  with  pilots  of  differing  ability,  must  still  be  conducted. 
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If  such  experiments  further  confirm  these  and  corollary  hypotheses) 
the  program  should  lead  to  more  efficient  performance  measurement  in  a 
wide  variety  of  aviation  and  space  tasks.  It  would  also  make  it  easier 
to  plan  for  performance  evaluation  in  future  man-machine  systems  for  which 
no  present  criteria  exist.  Finally)  it  would  have  important  implications 
for  training,  because  training,  or  at  least  a  good  portion  of  it,  could 
then  be  directed  to  the  very  heart  of  the  skill,  and  thus  lead  to  greater 
generality  of  transfer  of  training. 

George  C ha jet 

Project  Psychologist 

U.  S.  Naval  Training  Device  Center 
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BRIEF  OF  STUDY 

The  Phase  I  study  of  pilot  proficiency  (Krendel  and  Bloom,  1963) 
resulted  in  the  specification  of  aspects  of  behavior  unique  to  the  pilot  in 
a  manned  aircraft  system.  Rational  comparisons  of  systems  with  both  skilled 
and  unskilled  operators  supported  that  consistency  of  system  performance 
despite  changes  in  aircraft  or  pilot  characteristics  reflects  human  operator 
proficiency.  Overall  system  stability  appeared  to  be  dependent  upon  two 
characteristics  of  the  human  operator:  "adaptability"  and  "conservation  of 
effort".  These  concepts  imply  that  relatively  invariant  system  performance 
results  from  adap table  operator  performance  and  that  the  proficient  operator 
performs  under  a  minimum  enerpy  nude  of  operation. 

This  report  describes  a  series  of  laboratory  investigations  intended  to 
evaluate  "adaptability"  and  "conservation  of  effort"  as  parameters  of  track- 
inp  performance.  The  purpose  of  these  studies  was  to  evaluate  against  criteria 
of  system  performance  (a)  ability  of  the  human  operator  to  adapt  to  changes 
in  controlled  clement  dynamics,  (b)  ability  of  the  human  operator  to  perform 
under  different  rates  of  target  intcrmittency,  and  (c)  the  effects  of  track¬ 
ing  training  upon  these  abilities. 

Twenty-four  Us  of  known  levels  of  tracking  proficiency  and  experience 
were  required  to  perform  a  compensatory  tracking  task  first  under  conditions 
of  changing  controlled  element  dynamics  and  uhen  under  conditions  of  inter¬ 
mittent  target  presentation. 

The  effects  of  systematic  changes  in  controlled  element  dynamics  and 
target  (or  display)  intcrmittency  upon  operator  performance  were  measured  and 
analyzed  in  a  variety  of  ways.  In  all  cases  where  stable  measures  were  ob¬ 
tained,  the  proficient  trackers  retained  their  superiority  over  the  poor 
ones.  In  the  case  of  one  measure,  the  proficient  trackers  suffered  less 
decrement  in  performance  as  a  result  of  changing  dynamics  than  did  the  poor 
trackers,  but  in  all  other  measures,  the  decrements  due  to  changing  dynamics 
or  intcrmittency  were  the  same  for  the  proficient  and  the  poor  trackers. 
Additional  training  had  no  effect  upon  performance  under  any  of  the  experi¬ 
mental  conditions.  The  results  support  the  hypothesis  that  "adaptability" 
and  "conservation  of  effort"  are  important  parameters  of  system  performance 
in  that  the  proficient  operator  exhibits  adaptive  behavior  and  economy  of 
effort  to  a  greater  degree  than  the  non-proficient  operator.  The  results 
also  show  that  these  two  measures  are  not  correlated  and  therefore  tap  inde¬ 
pendent  aspects  of  proficiency. 

It  must  be  emphasized  that  tliis  was  the  first  step  in  the  measurement 
of  adaptability  and  economy  of  effort  as  understood  within  the  context  of 
servomechanism  theory,  and  that  more  refined  ways  of  obtaining  these  criteria, 
both  in  pure  tracking  type  tasks,  and  in  tasks  with  simulated  aircraft  instru¬ 
ments  must  be  developed  and  validated  on  pilots  of  known  caliber  before  the 
practical  role  of  these  concepts  can  be  folly  evaluated. 
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INTRODUCTION 


Background . 

This  program  is  concerned  with  the  development  and  measurement  of  flying 
skills.  In  particular,  we  are  concerned  with  perceptual  motor  skills  rather 
than  the  various  procedural  and  Judgmental  skills  which  a  Naval  Aviator  must 
possess.  It  is  the  first  experimental  outgrowth  of  USNTDC  Contract  No. 

N  61339-323,  Phase  I,  Study  of  Computer  for  Trainee  Performance  Evaluation. 

The  Phase  I  study  is  summarized  in  Technical  Report:  NAVTRADEVCEN  323-1, 

The  Natural  Pilot  Model  for  Flight  Proficiency  Evaluation  (Krendel  and  Bloom, 
1963).  The  Phase  1  program  exhaustively  examined  the  history  and  results  of 
the  various  subjective  and  objective  methods  for  measuring  and  improving 
piloting  skills.  In  general,  the  results  of  the  various  sophisticated 
psychological  examinations  of  the  training  process  did  not  produce  useful 
insights  into  the  dynamics  of  the  behavior  patterns  which  characterize  the 
piloting  of  an  aircraft. 

Areas  peripheral  to  training,  however,  bring  out  the  following  salient 
reference  points  for  our  development.  First,  the  day-to-day  effectiveness  of 
the  Landing  Sij;nal  Officer  in  making  his  split-second  evaluations  of  piloting 
performance,  which  he  bases  on  a  complex  of  subjective  factors  associated 
with  each  pilot,  makes  it  clear  that  subjective  evaluations  by  skilled  and 
insightful  observers  have  a  great  deal  of  operational  meaning.  Second,  the 
engineering  applications  of  test-pilot  opinion  rankings  of  aircraft  handling 
qualities  to  aircraft  design  further  serves  to  validate  the  effectiveness  of 
the  judgments  made  by  skilled  observers  in  a  flight  task.  Third,  there  exists 
a  well-developed  theoretical,  and  experimentally  verified,  correlation  between 
the  effective  dynamics  of  the  aircraft  under  evaluation  and  the  expert  pilot's 
judgments  of  handling  qualities.  Fourth,  there  is  a  large  and  detailed  body 
of  knowledge  relating  a  servomechanism  description  of  the  closed  loop  control 
behavior  of  the  pilot  to  the  effective  dynamics  of  the  vehicle  which  he  is 
flying . 


The  skeleton  of  the  approach  can  now  be  seen.  Subjective  evaluations 
of  piloting  ability  are  both  valid  and  valuable,  but  can  we  extract  a  measur¬ 
able  characteristic  of  piloting  behavior  which  correlates  with  the  subjective 
evaluation  and  which  is  susceptible  to  improvement  by  specific  training  tech¬ 
niques?  By  analogy  with  handling  qualities  theory,  this  characteristic  should 
be  the  description  of  what  the  pilot  has  to  do  in  a  servomechanisms  sense. 

In  other  words,  what  type  of  control  device  mist  the  pilot  be  in  order  to  fly 
at  given  levels  of  skill? 

Let  us  examine  motor  skills,  first  intuitively,  and  then  in  fine  grain 
in  a  servomechanism  context.  Skilled  behavior  is  characterized  first  and 
foremost  by  its  regularity  -  it  is  a  process  which  is  stable  in  the  statis¬ 
tical  sense.  The  execution  of  a  well-timed  lunge  or  parry  in  fencing, 
catching  a  fly  ball,  or  a  near-perfect  carrier  landing,  are  not  rare  events 
when  performed  by  a  skilled  man.  They  are  representative  samples  of  his  per¬ 
formance  repertory.  Stability  of  this  sort,  or  as  Krendel  and  Bloom  denoted 
it,  consistency  of  performance,  to  be  useful,  mist  be  maintained  over  time. 

As  time  passes,  however,  the  problem  may  change.  Wind  gusts  may  displace  the 
fly  ball,  the  aircraft  may  be  damaged  in  combat,  or  the  man  himself  may  change 
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through  fatigue  or  injury.  The  skilled  man  must  adapt  his  control  behavior 
to  both  these  external  or  internal  changes  in  dynamics.  Also,  his  perform¬ 
ance  must  be  relatively  stable  in  time,  even  if  the  operator  must  engage  in 
additional  tasks  that  might  suddenly  be  required;  hence,  some  sort  of  least 
effort  principle  of  control  behavior  must  obtain  for  skilled  behavior. 
Intuitively,  the  skilled  performer  makes  the  task  look  effortless,  graceful, 
and  easy.  The  human  servomechanism  then  must  be  characterized  by: 

1.  Adaptability  -  the  ability  to  generate  adaptive  behavior  as  a 
function  of  the  changing  dynamic  requirements. 

2.  Economy  of  effort  -  the  ability  to  use  energy  resources  efficiently 
so  that  motions  are  not  wasted  and  performance  can  be  sustained. 

The  problem  now  becomes  one  of  identifying  and  selecting  effective 
measures  of  adaptability  and  of  economy  of  effort  and  then  correlating  these 
measures  with  levels  of  skill. 

Approach. 

In  this  study  we  measure  adaptability  and  economy  of  effort  by  inference 
only.  The  reason  is  that  direct  measurement  of  either  human  dynamics  or  human 
metabolic  activity  involves  equipment  beyond  the  scope  of  this  program. 
Inferential  measurements  suffer  from  a  certain  lack  of  resolution;  however, 
they  have  the  advantage  of  being  simple  and  easily  obtained  under  many  experi¬ 
mental  conditions.  The  assessment  of  motor  skill  provides  somewhat  of  a  pro¬ 
blem,  for  the  skills  in  which  we  are  interested  in  the  case  of  Naval  pilots 
are  those  which  distinguish  the  "natural"  from  the  "mechanical"  pilot.  These 
are  the  skills  which  are  difficult  to  acquire  and  also  to  evaluate  by  con¬ 
ventional  atomistic  methods,  and  are  the  ones  that  play  a  vital  role  in 
differentiating  between  highly  successful  and  poorer  pilots.  We  do  not  know 
to  what  extent  they  are  related  to  the  so-called  innate  skills  that  aptitude 
tests  for  pilot  selection  tend  to  measure,  and  to  what  extent  they  are  inci¬ 
dentally  acquired  in  the  course  of  flying.  Put,  based  on  the  work  of  the 
first  phase  of  this  program,  the  hypothesis  is  put  forth  that  the  effective 
measurement  of  such  skills  will  get  at  the  very  heart  of  flying  proficiency 
and  will  have  greater  practicality  and  relevancy  because,  among  other  reasons, 
it  provides  a  global,  rather  than  atomistic  measurement. 

The  experimental  program  which  this  report  describes  is  s  trial  examina¬ 
tion  of  the  criteria  suggested  in  the  Phase  I  report.  The  subjects  were  college 
students  participating  in  a  tracking  task  which  bore  an  intuitively  satisfying, 
and  logically  defensible,  relationship  with  flight  control  tasks.  Subjects  were 
separated  into  two  distinct  groups  —  proficient  trackers  and  poor  trackers  ~ 
on  the  basis  of  the  skills  which  they  brought  to  the  task  from  the  very  begin¬ 
ning.  These  groups  were  exposed  to  the  task  and  the  two  inferentially  mea¬ 
sured  criteria  of  skill  were  obtained. 

The  first  part  of  the  major  experiment  consisted  of  presenting  a  sub¬ 
ject  with  a  set  of  changing  controlled  element  dynamics  in  which  he  was 
required  to  respond  in  such  a  manner  as  to  maintain  an  acceptable  level  of 
system  performance.  His  ability  to  maintain  consistent  system  performance 
under  conditions  of  changing  controlled  element  dynamics  was  taken  as  a 
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measure  of  his  ability  to  adapt.  The  second  part  of  the  major  experiment 
focused  on  economy  of  effort.  This  was  studied  in  terms  of  time  needed  to 
maintain  system  performance,  a  decreased  time  requirement  to  perform  a  task 
being  considered  characteristic  of  decreased  pilot  effort.  The  proficient 
operator  should  be  capable  of  recognizing  and  taking  advantage  of  the  coherence 
and  predictability  of  stimuli  and  of  the  relationship  between  his  control  and 
the  stimuli,  and  thus  he  should  require  less  time  and  less  information  to  per¬ 
form  his  task  than  the  non-proficient  pilot.  By  systematically  varying  the 
time  during  which  the  operator  is  permitted  to  view  the  stimuli,  it  is  possible 
to  evaluate  this  effect  in  relation  to  a  measure  of  system  performance.  In 
order  to  obtain  insight  into  the  effect  of  training  on  performance,  an  addit¬ 
ional  concern  under  both  experimental  situations  was  the  effect  upon  system 
performance  of  amount  of  previous  practice  on  the  task.  The  measured  index 
of  performance  was  RMS  error.  This  conventional  measure  of  error  is  parti¬ 
cularly  appropriate  in  this  case  because  it  emphasizes  large  departures  in 
performance  from  a  base  level. 
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EXPERIMENTAL  PROGRAM 

The  experimental  program  consisted  of  two  preliminary  studies  which 
established  levels  of  difficulty  of  controlled  element  dynamics,  and  useful 
rates  of  target  intermittency;  and  of  two  main  experiments  directed  toward 
the  establishment  of  "adaptability"  and  "time  in  control"  criteria  (Experi¬ 
ments  I  and  II). 

Apparatus . 

The  apparatus  used  in  the  preliminary  studies  and  the  subsequent  experi¬ 
ments  was  an  analog  computer  compensatory  tracking  device  (Figure  1).  A 
schematic  diagram  of  the  apparatus  is  given  on  page  30  of  the  Appendix.  The 
target  consisted  of  a  spot  of  light,  approximately  1  mm.  in  diameter,  displayed 
on  the  face  of  an  oscilloscope.  A  set  of  cross  hairs  was  used  to  indicate  the 
null  (zero  error)  position.  Target  motion  was  horizontal  and  was  furnished  by 
a  random  noise  generator  (Barnes,  1955).  The  noise  source  was  set  to  produce 
a  target  motion  of  bandwidth  two  radians/second.  The  tracking  control  device, 
a  rotary  control,  consisted  of  a  freely  turning  knob  and  a  pointer  mounted  on 
a  disk  (Figure  1).  A  clockwise  rotation  of  the  pointer  moved  the  target  from 
left  to  right.  The  absolute  value  of  the  error,  and  the  integrated  error 
squared  were  recorded  by  means  of  Ester line  Angus  recorders. 

5  sat  approximately  28  inches  from  the  face  of  the  scope  which  was  nor¬ 
mal  to”Yii s  line  of  vision.  The  tracking  control  device  was  mounted  on  an  arm 
rest  which  inclined  at  an  angle  of  20  degrees.  S  sat  in  a  conventional  class¬ 
room  chair  modified  to  accomodate  the  control  device. 

Preliminary  Study  1 


Subjects. 

Six  male  members  of  the  staff  of  the  Franklin  Institute  Laboratories 
served  as  Ss.  Each  subject  had  about  30  minutes  of  tracking  experience  in  a 
previous  exploratory  study  which  had  been  designed  to  select,  from  among  many 
possibilities,  the  most  promising  sets  of  control  dynamics  for  this  study. 

Procedure . 

Ss  were  required  to  perform  a  compensatory  tracking  task  under  two  forms 
of  controlled  element  dynamics:  Yc  =  Kc  and  Yc  =  Kc/s  where  Kc  3  2,  10,  15 
and  l/s,  5/s,  15/s,  and  30/s  respectively. 1  Ss  were  briefly  instructed  in  the 
operation  of  the  device.  Each  S  tracked  under  all  conditions,  the  total 
tracking  time  for  each  condition  being  three  minutes.  For  each  S,  the  order 
of  presentation  of  the  tacks  was  randomized  with  a  three  minute  rest  interval 
between  each  task. 


Results 

The  score  for  each  condition  was  taken  as  the  RMS  error  in  inches  on  the 
face  of  the  scope  over  the  last  minute  of  tracking.  The  means  and  standard 

1  Sec  glossary  (p  36-38)  for  explanation  of  controlled  element  dynaaics. 
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deviations  of  these  data  are  presented  in  Table  1. 

Table  1 


Means  and  Standard  Deviations 
of  RMS  Error  for  Controlled 
Element  Dynamics 


Conti-oiled  Element 


Dynamics  (Yc) 

M 

SD 

2 

0.25 

0.07 

10 

0.23 

0.03 

15 

0.26 

o.ou 

1/s 

0.39 

0.06 

5/s 

o.UU 

0.03 

15/s 

0.U8 

o.ou 

30/s 

O.UU 

o.ou 

The  analysis  of  variance  of  the  data  is  summarized  in  Table  2.  The  F  of  67.00 
for  controlled  element  dynamics  was  significant  at  less  than  the  .01~level. 

Table  2 

Analysis  of  Variance  for 
Controlled  Element  Dynamics 


Source 

df 

MS 

F 

Dynamics  (D) 

6 

0.067 

67.OO** 

Subjects  (S) 

5 

0.002 

2.00 

D  x  S 

30 

0.001 

^Significant 

at  less  than 

.01  level 

Duncan  Multiple  Range  Tests  (Duncan,  1 955")  of  the  difference  between  the  means 
were  performed  to  determine  which  of  these  control  dynamics  differed  signifi¬ 
cantly  from  each  other  in  RMS  error.  The  means  of  three  control  dynamics, 

Yc  =  10,  Yc  =  l/s,  and  Yc  =  15/s  were  found  to  be  significantly  different  from 
each  other  at  the  .01  level.  Thus,  three  difficulty  levels,  easy,  intermediate, 
and  difficult  were  established  for  use  in  the  major  experiment. 
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Preliminary  Study  2 


Subjects . 

Six  male  members  of  the  staff  of  the  Franklin  Institute  Laboratories 
served  as  Ss.  These  subjects,  just  as  the  Ss  in  the  first  preliminary  study, 
had  participated  in  an  exploratory  study  to”select  promising  sets  of  control 
dynamics,  and  thus  had  about  30  minutes  of  tracking  experience.  Ss  had  been 
assigned  to  one  or  the  other  of  the  two  preliminary  studies  on  a  random  basis. 

Procedure . 

So  were  required  to  perform  the  compensatory  tracking  task  under  eight 
conditions  of  tar-get  intermittency .  A  blanking  generator  described  in  the 
appendix  provided  an  intermittent  target  signal.  Target  on  time  was  maintained 
at  0.25  seconds  as  suggested  by  Lindquist  and  Gross  (1958)  and  off  time  was 
varied  between  0.25  and  lr.?5  seconds  (Table  3).  The  fraction  of  "off  time" 
per  cycle  was  defined  as 


Time  off  (sec.) 

Time  off  (sec.)  +  .25  (sec.) 

Controlled  element  dynamics  were  maintained  at  the  intermediate  level  of  diffi¬ 
culty,  Yc  =  1/s,  established  in  the  first  preliminary  study.  After  being 
instructed  in  the  operation  of  the  apparatus,  Ss  tracked  for  three  minutes 
under  each  condition.  The  order  of  presentation  of  the  conditions  was  random¬ 
ized  with  a  three  minute  rest  period  between  each. 

Table  3 


Conditions  of  Target  Intermittency 


Time  On 
(sec.) 

Time  Off 
(sec. ) 

%  of  Total 
Time  Off 

0.25 

lj.75 

95 

0.25 

2.25 

90 

0.25 

1.U2 

85 

0.25 

1.00 

80 

0.25 

0.58 

70 

0.25 

0.38 

60 

0.25 

0.25 

50 

1.00 

0.00 

0 
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Results 

System  performance  was  expressed  as  the  RMS  error  in  inches  over  the 
last  minute  of  tracking.  The  mean  RMS  error  in  inches  for  each  condition  of 
intermittency  is  shown  in  Table  b-  The  means  were  compared  in  the  Treatments 
x  Subjects  analysis  of  variance  summarized  in  Table  5- 

Table  b 


Means  and  Standard  Deviations 
of  RMS  Error  for  Intermittency 


$  of  Time  Off 

M 

SD 

95 

0.60 

0.07 

90 

0.33 

o.ob 

83 

o.3o 

0.0b 

80 

0.L6 

0.0b 

70 

0.h2 

0.05 

60 

0.b3 

0.02 

30 

0.b5 

0.0b 

0 

0.39 

0.03 

Table  3 

Analysis  of  Variance 
for  Target  Intermittency 

Source 

df  MS 

F 

Intermittency  (I) 

7  0.026 

13.00** 

Subjects  (S) 

3  0.002 

1.00 

I  x  S 

35  0.002 

Significant  at  less  than  .01  level 


The  F  of  13.00  for  target  intermittency  rates  was  significant  at  less  than 
the  701  level.  Differences  between  the  means  were  assessed  by  Duncan  Multiple 
Range  Tests  (1955)  in  order  to  establish  distinct  levels  of  difficulty.  The 
means  for  three  levels  of  intermittency  were  found  to  be  different  from  each 
other  at  well  beyond  the  .01  level  of  statistical  significance.  These  were 
the  target-off  times  of  )%,  85$,  and  60$,  respectively,  which  later  served 
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as  the  difficult,  intermediate,  and  easy  intermittency  tasks  of  the  second 
major  experiment. 


The  Major  Experiment 

Experiments  I  and  II  were  conducted  with  the  framework  of  a  single 
experimental  design.  The  purpose  of  these  studies  was  to  evaluate  against 
criteria  of  system  performance  (a)  ability  of  the  human  operator  to  adapt  to 
changes  in  controlled  element  dynamics  (Experiment  I),  (b)  ability  of  the 
human  operator  to  perform  under  different  rates  of  target  intermittency 
(Experiment  II),  and  (c)  the  effects  of  amount  of  previous  operator  exposure 
to  the  task  upon  these  criteria  (Experiments  I  and  II). 

Table  6  summarizes  the  experimental  design.  Ss  task  consisted  of  com¬ 
pensatory  tracking  under  conditions  of:  (a)  shift  in  control  sensitivity 
(Experiment  I)  and  (b)  intermittent  target  presentation  (Experiment  II). 

Prior  to  the  experimental  sessions,  Ss  were  screened  and  divided  into  two 
groups,  good  trackers  (proficient)  and  poor  trackers  (non-proficient),  and 
were  subjected  to  one  of  two  levels  of  experience  with  the  task. 

The  same  Ss  were  used  in  each  of  two  experimental  periods.  The  experi¬ 
mental  sessions~occurred  one  week  after  the  preliminary  screening.  Ss  per¬ 
formed  the  tracking  task  under  conditions  of  change  in  control  sensitivity 
during  the  first  period  (Experiment  I)  and  under  conditions  of  target  inter¬ 
mittency  during  the  second  period  (Experiment  II).  Trial  and  sequence  effects 
were  controlled  by  presenting  the  tasks  as  prescribed  by  a  3  x  3  Latin  square. 

Method 


Apparatus . 

The  apparatus  was  the  same  as  used  in  the  preliminary  studies.  (See 
pages  29-30  in  the  Appendix.)  Changes  in  controlled  element  dynamics  were 
acconplished  by  an  automatic  switching  mechanism  which  changed  amplifier 
gains  and  feedback  at  specified  time  intervals ;  target  intermittency  was 
achieved  by  a  blocking  oscillator  and  a  pulse  width  control. 

Subjects . 

Thirty-six  paid  volunteer  male  undergraduate  students  from  local  uni¬ 
versities  served  as  Ss.  None  of  the  Ss  had  previous  tracking  experience .  On 
the  basis  of  preliminary  screening,  tKis  group  was  reduced  to  2li  Ss. 

Preliminary  Session. 

The  proficiency  of  Ss  as  measured  by  RMS  error  over  the  last  minute  of 
tracking  was  assessed  in  a  preliminary  session  during  which  they  were  engaged 
in  three  minutes  of  compensatory  tracking.  During  this  session,  control 
sensitivity  was  at  the  intermediate  level^  Yc  =  l/s.  On  the  basis  of  per¬ 
formance  on  this  task,  Ss  were  divided  into  two  highly  distinct  groups,  good 

Based  on  the  first  preliminary  study. 
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Experimental  Design 


Proficiency  Experience  Order 


Experiment  I 
Trial 

1  ?  3 


Experiment  II 
Trial 

12  3 


Subject 


'Latin  letter:;  A,  B,  and  C  ;=  three  level:;  of  change  in  controlled  element  dynamics 
D,  E,  and  V  ~  three  level:;  of  intermittency. 
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trackers  and  poor  trackers.  The  pood  trackers  consisted  of  the  12  Ss  with  the 
lowest  RMS  error;  the  poor  trackers  consisted  of  the  12  Ss  with  the~highest 
RMS  error.  The  remaining  Ss  were  dropped.  The  difference  in  mean  RMS  error 
between  the  two  groups  was  sipnificant  at  less  than  the  .001  level  (t  =  6.?6, 
df  =  22). 

Trackinp  Experience . 

In  order  to  introduce  prior  traininp  as  an  independent  variable,  half  the 
pood  trackers  and  half  the  poor  trackers  received  an  additional  12  minutes  of 
experience  with  the  compensatory  trackinp  task  before  the  experimental  session. 
The  remaininp  Ss  received  no  additional  experience.  During  this  session,  con¬ 
trol  sensitivity  apain  was  at  the  intermediate  level  of  difficulty!  Yc  =  l/s. 

Experimental  Session  -  Experiment  I. 

Experiment  I  dealt  with  adaptability,  i.e.,  compensatory  tracking  under 
conditions  of  changing  controlled  element  dynamics.  There  wore  three  levels  of 
change  in  dynamic:;,  each  constituting  a  trackinp  task  lasting  six  minutes. 

Every  level  started  with  two  minutes  of  trackinp  with  Yc  =  l/s  and  ended  with 
two  minutes  of  Y„  -  l/s,  but  the  trackinp  during  the  third  and  fourth  minute 
was  performed  under  changed  control  sensitivity:  Yc  =  10  i'or  the  first,  Yc  = 

Z/s  i'or  the  second,  and  Yc  -  l'J/s  for  the  third  level.  The  following  tabulation 
summarizes  these  levels  or  tasks: 

First  Two  Minutes _ Second  Two  Minutes _ Third  Two  Minutes 

1.  level  Yc  =  l/s  Yc  =10  Yc  =  l/s 

2.  level  Yc  =  l/s  Yc  =  37 s  Yc  =  l/s 

3.  level  Yc  =  l/s  Yc  •-  lZ/s  Yc  =  l/s 

Thus,  at  the  end  of  the  second  minute,  anti  at  the  end  of  the  fourth  min¬ 
ute  a  change  of  dynamics  took  place  on  each  level.  Since  Yc  =  10  had  been  found 
in  the  preliminary  study  to  be  easy,  Yc  =  Z/s  intermediate,  and  Yc  =  lZ/s  diffi¬ 
cult,  the  three  levels  afforded  an  opportunity  to  study  the  effect  of  the  diffi¬ 
culty  of  the  inserted  task  upon  the  subjects'  reaction  to  the  change. 

The  subjects  were  not  informed  that  a  change  of  dynamics  (control-display 
relationships)  would  occur  during  this  experiment. 

Experimental  Session  -  Experiment  II. 

The  second  experimental  session  occurred  30  minutes  after  the  first. 

During  this  session,  Ss  were  required  to  perform  the  compensatory  tracking 
task  under  each  of  the  three  levels  of  target  intermittency,  easy,  inter¬ 
mediate,  and  difficult,  shown  in  Table  7.  Controlled  element  dynamics  was 
maintained  at  the  intermediate  level  of  difficulty,  Yc  =  l/s.  Prior  to  the 
experimental  session,  Ss  were  informed  that  the  target  would  appear  inter¬ 
mittently.  They  were  permitted  to  observe  the  behavior  of  the  target  and  to 
attempt  to  track  under  these  conditions.  The  total  tracking  time  under  each 


lased  on  the  first  preliminary  study. 
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condition  war.  six  minutes. 

Tabic  7 

Conditions  <>f  Target  lntermittency 


Time  On 
(sec.) 

Time  Off 
(sec.) 

l  of  Total 
Time  Off 

Level""' 

0 .  :_’3 

Ji 

3  3 

Diff icult 

0.2,'J 

l.lik 

Intermediate 

0 . 2[> 

0.38 

00 

Easy 

"Based  upon 

results  from 

Experiment  II 

Results 


Experiment.  I. 


To  assess  the  effect,  upon  system  per  To  nuance  of  Cs  response  to  chancing 
controlled  element  dynamics,  a*  error  acore  ;uid  a  time  acore  were  obtained  at 
each  level  of  chance.  The  error  acore  war  taken  ar  the  difference  between 
RMS  error  (in  inchea)  meant  red  over  a  half-minute  interval  before  chance  in 
dynamics  and  the  RFC  error  neamred  over  a  half-minute  interval  after  change 
in  dynamica.  There  c< natituted  the  intervale  between  1.3  and  2.  minutes ,  and 
between  )j  and  It. 3  minutes  tracking  for  each  of  the  three  tanks  or  levels . 
Since,  aa  war  pointed  out.  above  (p  li),  the  first  2  minuter  and  the  last  2 
minuter  of  each  of  t.he  '"-minute  tracking  t.aaka  were  performed  at  Yc  =  l/s,  all 
acorea  arc  baaed  on  Yc  -  l/a  only.  The  time  acore  war  the  time  in  seconds  to 
return  to  and  maintain  for  15  or  more  seconds  an  absolute  error  score  equal 
to  or  leas  than  a  base  line  plus  or  minus  one  standard  deviation.  The  base 
line  was  defined  as  the  mean  level  of  system  performance  (i.e.,  mean  level  of 
absolute  value  of  the  error)  over  the  30  seconds  prior  to  the  first  change  in 
dynamics.  Measures  were  taken  every  1.25  seconds. 

The  means  and  standard  deviations  of  the  time  and  error  scores  are  pre¬ 
sented  in  Tables  8  and  <  respectively.  The  means  of  these  measures  were  com¬ 
pared  by  a  five  classification  replicated  Latin  square  analysis  of  variance 
(Tables  10  and  11). 

Analysis  of  variance  (Table  10)  with  difference  in  RMS  error  before  and 
after  change  of  dynamics  as  the  measure  of  system  performance  resulted  in  a 
F  of  h.)y  for  proficiency  significant  at  less  than  the  .03  level.  In  general, 
The  smallest  differences  in  RMS  error  were  obtained  for  those  systems  contain¬ 
ing  proficient  trackers.  Since  most  of  the  differences  were  in  the  direction 
of  increased  error,  it  can  he  concluded  that  the  poor  trackers  revealed  not 
only  greater  variability  but  a  greater  decrement  as  a  result  of  change  in 
dynamics  than  did  the  proficient  trackers.  The  F's  for  experience  and  the 
Dynamics,  x  Proficiency  interaction  were  not  significant.  Nor  were  any  of 
the  remaining  main  effects,  or  interactions  significant.  Thus  we  can  con¬ 
clude  that  the  greater  performance  variability  and  decrement  of  the  poorer 
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Based  on  original  data.  Means  and  standard  deviations  of  the  transformed  scores  appear  in  Table 
Appendix,  page  32. 
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Table  10 


Analysis  of  Variance  for  Controlled  Element  Dynamics: 

Difference  in  RMS  Error  30  Seconds  Before  and  After  Change  in  Dynamics  1) 


Source 

df 

MS 

F 

Between  Subjects 

23 

Proficiency  (P) 

1 

0.0003U9 

U.99* 

Training  (T) 

1 

0.000001 

Sequence  (S) 

2 

0.000253 

3.61 

P  x  T 

1 

0.000085 

1.21 

P  x  S 

2 

0.000015 

T  x  S 

2 

0.000068 

P  x  T  x  S 

2 

0.000103 

1.1:7 

Error 

12 

0.000070 

Within  Subjects 

m 

Dynamics  (D) 

2 

0.000078 

2.05 

Trials  (Tr) 

2 

0.000093 

2.15 

D  x  P 

2 

0.000017 

D  x  T 

2 

0.000085 

2. 21* 

D  x  P  x  T 

2 

0.000081 

2.13 

Tr  x  P 

2 

0.000006 

Tr  x  T 

2 

0.000117 

3.08 

Tr  x  P  x  T 

2 

0.00003? 

Error 

32 

0.000038 

^Significant  at  less  than  .05  level. 

^Based  on  transformed  data. 
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Table  11 


Analysis  of  Variance  for  Controlled  Element  Dynamics: 
Time  to  Return  to  Base  Line  1) 


Source 

df 

MS 

F 

Between  Subjects 

23 

Proficiency  (P) 

1 

o.o5h3 

Training  (T) 

1 

0.00U9 

Sequence  (S) 

2 

0.1550 

P  x  T 

1 

0.U278 

P  x  S 

2 

0.53149 

T  x  S 

2 

2.0156 

2.1l* 

P  x  T  x  S 

2 

0.0U41 

Error 

12 

0.9h06 

Within  Subjects 

h8 

Dynamics  (D) 

2 

0.8287 

Trials  (Tr) 

2 

0.9679 

D  x  P 

2 

1.5926 

2.60 

D  x  T 

2 

0.1325 

D  x  P  x  T 

2 

0.5219 

Tr  x  P 

2 

0.1750 

Tr  x  T 

2 

0.51514 

Tr  x  P  x  T 

2 

0.2235 

Error 

32 

0.6121 

Based  on  transformed  data. 
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trackers  was  true  regardless  ol'  level  of  change  in  dynamics  or  level  of  train¬ 
ing. 


To  further  assess  the  effect  of  changes  in  controlled  element  dynamics 
and  operator  proficiency  upon  system  performance,  the  mean  RMS  error  over  all 
levels1,  for  the  30  seconds  prior  to  change,  was  compared  with  the  mean  RMS 
error  over  all  levels1  for  the  first  30  seconds  after  the  second  change  (Yc  = 
l/s  in  both  c-'ses).  The  means  and  standard  deviations  for  each  level  of  pro¬ 
ficiency  appear  in  Table  12.  Analysis  of  variance^  resulted  in  F's  for  pro¬ 
ficiency  and  dynamics  significant  at  less  than  the  .01  level  (Table  13).  The 
Dynamics  x  Proficiency  interaction  was  not  si(?iificant.  For  both  the  profi¬ 
cient  and  non-proficient  Ss,  changing  controlled  element  dynamics  resulted  in 
a  decrement  in  system  performance.  The  magnitude  of  the  difference  between 
the  two  groups  for  both  the  pre-change  and  post-change  conditions  remained 
relatively  constant. 


With  the  time  score  as  the  measure  of  system  performance,  none  of  F's 
resulting  from  analysis  of  variance  were  significant1.  ~ 


^Differences  in  HMO  error  at  each  level  prior  to  pooling  were  not  significant 
(see  Appendix,  p  3l* )  • 


The  effects  of  training,  trials,  and  sequence  were  assumed  to  be  non-signifi¬ 
cant  as  in  the  previous  analysis . 

^The  variances  of  these  scores  were  very  high ,  and  the  observed  differences 
among  the  means  were  unrelated  to  any  of  the  variables.  To  determine  whether 
this  was  due  only  to  details  of  measurements,  the  time  score  as  measure  of 
system  performance  was  obtained  in  several  other  ways.  In  one  of  these,  we 
measured  time  to  return  to,  and  maintain  for  10  seconds  an  error  score  equal 
to  or  less  than  the  subject's  own  (rather  than  mean)  base  line  *  one  sigma. 
None  of  these  other  methods  of  obtaining  time  scores  yielded  significantly 
different  means,  or  smaller  variances  than  the  ones  reported  in  our  original 
measures. 

While  time  to  adjust  to  changing  control  dynamics  seemed  to  be  an  inportant 
aspect  of  adaptability  on  theoretical  grounds,  it  failed  to  stand  up  under 
this  particular  experimental  test.  It  may  be  that  other  variables  in  the 
task  masked  its  influence,  and  that  measuring  time-to-adjust  in  tasks  with 
simulated  aircraft  instruments  might  have  yielded  a  more  stable  measure  of 
adaptability. 
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Table;  I  ? 

Moan:;  and  Standard  Deviations  oi'  KMS  Error 
Before  and  After  Change  in  Dynamics 


Level  of  Change  of 
Controlled  Element 
Dynamics 

Level  of 

Proficiency 

Proficient 

Not  Proficient 

M  SD 

M 

SD 

H1 

0.31  0.0‘J 

o.ho 

0.06 

A2 

0.3)l  0.05 

0.1,3 

0.0J 

^Based  on  me.-in  HMS  error  over  all  levels 

for  tiie 

30  seconds 

prior  to  change. 

^Baaed  on  mean  HMS  error  over  aLl  love]  r, 

for  the 

30  seconds 

after  change. 

Table  13 

Analysis 

of  Variance  for  Dynamics 

Source; 

df 

MS 

F 

Between  Subjects 

?J 

Proficiency  (P) 

1 

0.0916 

19.91** 

Error 

?? 

0.00)i6 

Within  Subjects. 

■)U 

Dynamics  (D) 

1 

0.0080 

11.43** 

P  x  D 

1 

0.00?? 

Error 

22 

0.0007 

Significant  at  less  than  .01  level 
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Experiment  II. 

For  each  level  of  intermittency,  the  RMS  error  in  inches  over  the  last 
two  minutes  of  tracking  was  taken  as  the  measure  of  system  performance.  The 
means  and  standard  deviations  of  the  original  data  appear  in  Table  14.  Be¬ 
cause  of  the  presence  of  a  large  number  of  zero  scores  in  the  original  data, 
it  was  advisable  to  transform  the  scores  before  carrying  out  the  analysis  of 
variance  (Federer,  1955;  Kempthorne,  1952).  The  details  of  the  transformation 
and  the  transformed  means  and  standard  deviations  appear  in  the  Statistical 
Appendix.  Comparison  of  the  means  of  the  transformed  scores  was  accomplished 
by  a  five  classification  replicated  Latin  square  analysis  of  variance  (Table 
15).  The  analysis  resulted  in  an  F  of  134.91  for  target  intermittency  which 
was  significant  at  less  than  the  .01  level.  Increased  rates  of  target  inter¬ 
mittency  were  accompanied  by  increases  in  the  magnitude  of  the  RMS  error. 
Differences  between  the  three  rates  of  intermittency  were  significant  at  the 
.01  level  in  Duncan  Multiple  Range  Tests. 

The  F  of  7 .06  for  proficiency  wa:;  significant  at  less  than  the  .025  level 
Neither  the  main  effect::  of  training,  sequences,  or  triala  nor  any  of  the  inter 
actions  of  these  effect::  or  of  proficiency  and  training  wore  significant. 

The  relationship  between  Uie  means  for  proficiency  .and  level  of  inter¬ 
mittency  is  presented  in  Fixture  2.  Also  shown  is  the  base  point  obtained  in 
Experiment  I,  i.e.,  the  mean  RMS  error  over  all  levels  of  change  in  controlled 
element  dynamics  for  Ute  30  seconds  prior  to  change.  Systems  containing  pro¬ 
ficient  Ss  produced  less  RMS  error  than  those  with  non-proficient  Ss.  This 
relationship  remained  constant  l'or  all  levels  of  intermittency  and  was  not 
affected  by  training.  Ignoring  the  proficiency  of  the  Ss,  system  performance 
was  best  with  continuous  target  presentation  and  deteriorated  progressively 
with  target  off- tines  of  (•  0,  65,  and  75  per  cent. 

Relationship  Between  the  Measures. 

A  rank  difference  correlation  between  the  RMS  error  scores  for  controlled 
element  dynamics  and  the  RMS  error  scores  for  intermittency  resulted  in  a 
Spearman  Rho  of  .36.  The  Rho  failed  to  reach  significance  at  the  .05  level 
(t  -  1.73,  df  -  22).  A  Rho  of  .37  was  obtained  between  the  RMS  error  score 
and  the  time  score:  for  controlled  element  dynamics.  The  t<  of  1.77  was  not 
significant  at  the  .05  level  (df  =  22).  The  small  magnitude  of  the  Rho's  and 
their  failure  to  reach  significance  indicates  that  there  is  little  or  no 
relationship  between  the  measures. 
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i-*eans  and  standard  deviations  of  the  transfcrned  scores  appear  in  Table  3A,  Appendix 
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Tabic  lb 


Analysis  of  Variance  for  Target  Intermittency 


1 


Source 

df 

MS 

F 

Between  Subjects 

23 

Proficiency  (P) 

1 

0.01201 

7.06* 

Experience  (E) 

1 

0.00002 

Sequence  (S) 

2 

0.0000? 

P  x  E 

1 

0.00070 

P  x  S 

O 

0.00233 

1.38 

E  x  S 

o 

0.00027 

P  x  E  x  S 

') 

0.0012b 

Error 

1? 

0.00170 

Within  Subjects 

lib 

Intermittency  (I) 

2 

0.13761 

13b. ri** 

Trials  (Tr) 

O 

0.0007b 

I  x  P 

2 

0.0002b 

I  x  T 

( 

0.00116 

l.ib 

I  x  P  x  E 

0.0022b 

2.2b 

Tr  x  P 

n 

o.ooiby 

1.56 

Tr  x  E 

0.00100 

Tr  x  P  x  E 

r. 

0.00021 

Error 

32 

0.00102 

Based  on  transformed  data. 

■ft 

Significant  at  loss  than  .025  level. 

><-ft 

Significant  at  less  than  .01  level. 


GROUP  MEAN  RMS  ERROR  (in.) 
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%  OFF  TIME 


FIG.  2.  GROUP  MEAN  RMS  ERROR  AT  EACH  LEVEL  OP  TARGET 
INTERMITTENCY  FOR  OOOO  AND  FOR  POOR  TRACKERS 
(Tha  Zaro  Off  Tima  Racord  Comaa  From  Eiparimant  I) 
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General  Discussion. 


Systems  containing  proficient  trackers  displayed  higher  system  perform¬ 
ance  under  all  conditions  of  changing  controlled  element  dynamics  and  target 
intermlttency .  Considering  ability  to  function  under  these  conditions  as 
criteria  of  adaptability  and  conservation  of  effort  respectively,  it  was  found 
that  the  proficient  tracker  (a)  exhibits  adaptive  behavior  to  a  greater  degree 
than  non-proficient  trackers,  and  (b)  exhibits  a  greater  degree  of  conservation 
of  effort.  If  one  allows  extrapolation  from  the  known  to  the  unknown  "per¬ 
centage  off1  times  (Figure  2),  one  can  argue  that  the  poor  operator  group's 
performance  with  an  uninterrupted  display  was  equalled  by  that  of  the  good 
pilot's  with  a  70  per  cent  off  time.  In  other  words,  one  might  argue  that 
good  trackers  do  as  well  as  poor  trackers  with  only  30  per  cent  of  the  effort. 

Another  interpretation  of  the  effect  of  intermittency  is  possible.  Since 
a  high  level  of  veliicle  control  is  possible  under  high  levels  of  intermittency, 
it  follows  that  the  remaining  signal  contains  enough  information  for  such  con¬ 
trol.  What  is  lost  during  the  "off  time"  must  therefore  be  mainly  redundancy. 
When  the  residual  information  falls  below  some  value,  e.g.,  that  information 
displayed  at  about  70  per  cent  off  time,  performance  worsens  rapidly.  We 
conclude,  therefore ,  that  the  difference  in  rate  of  error  increment 

A  error 

A  intermittency 

between  good  and  poor  trackers  (Figure  2)  is  due  in  an  important  part  to  the 
efficiency  of  data  processing  (or,  in  other  terms,  to  a  tendency  on  the  part 
of  poor  trackers  to  rc:;pond  to  irrelevant  information  which  good  trackers 
ignore).  Another  way  of  saying  the  same  thing  is  that  poor  trackers  have 
something  like  a  built  in  source  of  noise  so  that  the  required  redundancy  for 
a  given  level  of  information  transmission  is  greatly  increased.  This  is  not 
an  unexpected  finding  and  it  finds  strong  support  from  various  physiological 
investigations.  First  of  all,  it  should  be  noted  that  no  response  of  an 
organism  ever  occurs  in  isolation.  There  are  always  a  number  of  simultaneous 
responses  occurring.  It  is  well  known  that  one  main  difference  between 
skilled  and  unskilled  performance  is  the  amount  of  tenseness  which  the  trainee 
exhibits.  Skilled  performance  is  characterized  by  general  relaxation.  Only 
those  muscle  groups  directly  involved  in  the  performance  are  active.  But  un¬ 
skilled  performance  is  different.  Beginners  are  tense  all  over  (literally). 
Unskilled  trackers  using  (say)  their  right  hands  will  show  significant  muscu¬ 
lar  action  potentials  in  their  left  big  toe.  This  tenseness  is  in  general 
facilitory  in  the  sense  that  it  reduces  reaction  time,  etc.  But  since 
inappropriate  responses  are  facilitated  as  much  or  more  than  appropriate 
response,  the  net  effect  is  a  reduction  in  efficiency.  These  inappropriate 
responses  are  equivalent  to  noise  in  an  information  sense.  Thus,  we  have  a 
basis  for  the  requirement  of  poor  trackers  for  a  greater  degree  of  redundancy. 

Use  of  the  tracking  task  in  the  present  investigation  was  based  upon  its 
importance  as  a  component  of  the  pilot's  task  and  its  general  acceptance  as  a 
criterion  of  pilot  j>crformancc.  Changing  controlled  element  dynamics  were  in¬ 
tended  to  simulate  changes  in  aircraft  dynamics  during  flight;  intermittency 
of  target  simulated  conditions  during  which  attention  might  be  diverted  (e.g., 
during  emergency,  combat,  or  routine  non-flight  tasks).  The  relationship 
between  ability  to  meet  these  two  types  of  conditions  and  system  consistency 
was  demonstrated. 
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Although  the  present  investigation  demonstrates  the  consistency  of  our 
measures  with  tracking  proficiency,  their  exact  role  in  predicting  differences 
in  proficiency  of  experienced  pilots  and  students  at  various  levels  of  training 
needs  to  be  established*  Rank  difference  correlations  between  the  RMS  error 
measure  of  3s  performance  under  continuous  tracking  conditions^-  and  the  mean 
error  scores  for  changing  controlled  element  cfynamics  and  intend. ttency  resulted 
in  Rhos  of  *U8  (df  -  22,  t  -  2*57)  and  *5U  (df  -  22,  t  -  3*01)  respectively, 
significant  at  less  than  the  .05  level*  These  relatively  low  correlations, 
when  considered  within  the  context  of  the  theory  and  experiment  of  this 
program,  suggest  an  important  implication  for  the  prediction  of  pilot  performance  t 
It  may  well  be  that  adaptability  and  economy  of  effort  test  aspects  of  flight 
performance  that  conventional  measures  have  never  tapped*  While  only  an 
experiment  with  pilots  of  exceptional,  high,  and  average  proficiency  can 
determine  the  appropriate  weights  that  should  be  assigned  to  these  two  measures, 
great  care  must  be  taken  that  the  differentiation  of  criterion  pilots  be  made 
on  the  basis  of  global  types  of  measures  (e.g*,  LSO  opinion  based  on  several 
carrier  landings)  rather  than  on  the  item-by-item  evaluation  of  conventionaal 
measurement  which  might  mask  the  role  of  "natural"  pilot  skills* 

The  lack  of  correlation  between  adaptability  and  conservation  of  effort, 
once  these  measures  have  been  validated  on  pilot  populations,  will  be  an  oper¬ 
ational  advantage.  Because  in  any  equation  for  the  prediction  of  performance, 
predictors  that  have  a  correlation  with  that  performance,  but  not  with  each 
other,  account  for  a  larger  a hare  of  the  predictive  effort. 

The  extra  tracking  experience  which  half  the  proficient  and  half  the  poor 
subjects  had  received  prior  to  the  main  experiment  had  no  effect  upon  system 
performance*  It  is  essential  to  note  that  what  is  called  in  this  experiment 
•training"  did  not  include  tracking  under  conditions  of  changing  dynamics  or 
intermittent  appearance  of  the  target,  but  merely  continuous  tracking  j  and 
that  none  of  the  subjects  received  detailed  instructions  concerning  the  nature 
of  tracking  undar  changing  or  intermittent  conditions*  Failure  to  obtain 
Improvements  in  adaptability  or  conservation  of  effort  should  therefore  not 
be  surprising.  However,  the  operational  situation  parallels  this,  lnamarch 
as  the  major  portion  of  training,  both  in  simulators  and  in  training  aircraft, 
is  more  related  to  this  study's  continuous  tracking  than  to  changing  dynasica 
or  in  tend. ttency ;  thus,  despite  extensive  training,  the  so-called  "natural" 
pilot  skills  are  probably  not  always  acquired  and  acme  pilots  become  "overloaded" 
whan  an  unusually  high  degree  of  adaptability  and  conservation  of  effort 
suddenly  become  essential  to  mission  success* 

TWo  major  research  Issues  will  have  to  be  resolved  before  direct  appli¬ 
cation  of  this  study's  criteria  can  bs  made  to  float  practice* 

1*  Adaptability  and  economy  of  effort  can  be  measured  by  tracking  tasks 
such  as  tbs  ones  used  In  this  study,  or  by  simulated  aircraft  instruments  modi¬ 
fied  to  permit  systematic  changes  in  caitrol-di splay  dynamics  and  intend  ttency. 
Which  of  these  two  means  of  measuring  adaptability  and  econcay  of  effort  ia  the 
more  efficient  predictor  of  overall  flying  proficiency? 


•LPre-ahift  base  line  of  Experiment  It. 
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2.  What  is  the  effect  of  extensive  practice  on  tasks  that  measure  our 
two  criteria,  upon  subsequent  flying  performance? 

It  must  be  admitted  that  research  to  answer  these  questions  is  difficult 
and  time-consuming  because  of  the  multitude  and  complexity  of  variables  that 
are  involved  in  motor-skill  and  transfer  of  training,  many  of  which  are  still 
unexplored.  But  once  the  most  efficient  measures  of  our  two  criteria  have  been 
identified,  great  payoffs  in  pilot  proficiency  measurement  and  training  will  be 
possible . 
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APPENDIX 

APPARATUS 

The  apparatus  consisted  of  an  analog  computer  and  a  tracking  device  com¬ 
posed  of  a  manually  operated  rotary  control  and  a  display  (Figure  l).  A 
schematic  diagram  is  given  in  Figure  3. 

The  tracking  control  arm  activated  transducers  for  electrical  take-off 
to  the  system.  The  dotted  lines  enclose  four  alternative  feedback  connections. 
Since  the  feedback  and  input  impedances  were  readily  changeable,  considerable 
flexibility  was  available.  Selection  of  a  particular  network  was  switch  con¬ 
trolled.  A  timing  device  permitted  automatic  switching  of  amplifier  gains 
and  feedback  at  uniform  time  intervals. 

Target  motion  was  furnished  by  a  random  noise  generator  (Barnes,  1955). 

A  switch  operated  filter  selected  target  motion  band  widths.  The  loop  was 
completed  by  a  summing  amplifier  and  a  display  consisting  of  a  standard  labor¬ 
atory  oscilloscope . 

The  "blanking  generator"  consisted  of  a  blocking  oscillator  of  variable 
frequency  (a  Hewlett  Packard  Model  202A  Low  Frequency  Function  Generator  was 
used  for  this  purpose)  and  a  pulse  width  control.  Width  control  allowed 
adjustment  of  the  on-off  ratio. 

Error  voltage  was  fed  through  a  squaring  circuit  and  integrated  (cir¬ 
cuits  not  shown) .  Integrated  error  squared  was  recorded  by  means  of  Esterl’ne 
Angus  Recorders.  Paper  speed  was  set  at  twelve  inches  per  minute. 

With  the  exception  of  the  display,  tracking  device,  and  variable  fre¬ 
quency  generator,  the  components  were  assembled  in  a  single  19  inch  rack. 

Experiments  I  and  II. 

For  both  the  time  and  the  error  scores,  the  means  and  the  variances  were 
proportional.  The  intermittency  data  contained  a  number  of  zero  scores.  It 
was  therefore  advisable  to  transform  the  data  prior  to  the  execution  of  the 
analysis  of  variance .  The  most  appropriate  transformation  was  found  to  be 
Log10  +  1,  for  the  time  data,  Log^Q  X  +  5,  for  the  error  data,  and  /X  +  .5 
for  the  intermittency  data  (Federer,  1955;  Kempthome,  1952).  The  means  and 
standard  deviations  of  the  transformed  scores  are  presented  in  Tables  1A,  2A, 
and  3A.  A  replicated  five  classification  Latin  square  analysis  of  variance 
was  applied  to  the  transformed  scores  of  each  measure  (Tables  12,  13,  and 
15).  The  means  and  standard  deviations  of  Tables  10,  11,  and  lh,  are  in 
terms  of  the  original  (non-transformed)  measures. 

Tables  UA  and  6A  present  the  means  and  standard  deviations  for  each 
level  of  change  in  controlled  element  dynamics  for  the  30  seconds  prior  to 
change  and  the  30  seconds  after  change  respectively.  The  analysis  of  var¬ 
iance  for  these  means  appear  in  Tables  5A  and  7A. 
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Table  liA 


Means  and  Standard  Deviations  of  RMS  Error  for  Each  Level  of 
Controlled  Element  Dynamics  l'or  the  30  Seconds  Prior  to  Change 


Level  of  Change 
of  Controlled 

Element  Dynamics 

Level 

of  Proficiency 

Proficient 

Not  Proficient 

M 

SD 

M 

SD 

1 

0.31 

0.05 

0.38 

0.06 

2 

0.31 

o.oli 

O.hl 

0.0? 

3 

0.3? 

0.06 

o.ho 

0.05 

Table 

_5A 

Analysis 
Over  All 

of  Variance 
Levels  for 

for  Dynamics: 
the  30  Seconds 

Mean  RMS  Error 
Prior  to  Change 

Source 

df 

MS 

F 

Between  Subjects 

23 

Proficiency  (P) 
Error 

1 

22 

0.12W4 

0.01hl 

8.82** 

Within  Subjects 

m 

Level  (L) 

L  x  P 

Error 

2 

2 

1*1* 

0.0018 

0.0026 

0.0012 

1.50 

2.17 

Significant  at  less  than  .01  level. 
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Table  6A 

Means  and  Standard  Deviations  of  RMS  Error  for  Each  Level  of 
Controlled  Element  Dynainic:;  for  the  30  Seconds  After  Change 


Level  of  Proficiency 

Level  of  Change 
of  Controlled 

Element  Dynamics 

Proficient 

Mot 

Proficient 

M 

SD 

M 

SD 

1 

0.3b' 

0.06 

O.Iili 

0.08 

O 

c 

0.  32 

o.ob 

0.L|2 

0.08 

3 

o.'iU 

0.06 

0.1*2 

0.0b' 

Analysis 
Over  All 

Table  7A 

of  Variance  for  Dynamics: 
Levels  for  the  30  Seconds 

Mean  RMS  Error 
After  Change 

Source 

df 

MS 

F 

Between  Subjects 

23 

Proficiency  (P) 

1 

o.ibil 

10.72 

Error 

22 

0.0141 

Within  Subjects 

48 

Dynamics  (D) 

2 

0.0032 

1.03 

P  x  D 

2 

0.0001 

Error 

44 

0.0031 

* "Significant  at  leas  than  .01  level. 
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Compensatory  Tracking 


RMS  Error 

Absolute  Value  of 
the  Error 

radians/second 
Band  Width 

Controlled  Element, 

Y 

c 

K 

c 

Target  Intermittency 

Transfer  Function 


GLOSSARY 


The  operator  is  presented  with  an  input  (display)  con¬ 
sisting  of  an  indicator  showing  the  difference,  or 
error,  £(t),  between  the  forcing  function  i(t),  and 
the  system  output  r(t).  The  operator's  task  is  to 
minimize  the  error  signal  presented  by  trying  to  keep 
a  dot,  superimposed  on  a  stationary  target. 

Root  Mean  Square  Error: 


Error  Voltage  recorded  without  regard  to  sign. 

A  unit,  for  the  measurement  of  frequency;  6.28  radians 
equal  1  cycle;  1  radian  equals  'J7 . 3° • 

The  width  of’  the  resonant  curve,  in  frequency  units, 
at  the  point  at  wliich  the  power  in  the  circuit  is  one- 
hall'  of  the  maximum  power  at  resonance,  expressed  as 
a  percentage  of  the  resonant  frequency. 

-  The  dynamics,  of  the  type  of  element  being  controlled 
by  the  operator,  such  as  an  aircraft,  automobile,  etc., 
and  the  actual  means  of  exerting  control  :mch  as  a 
stick  or  wheel  with  their  associated  restraints 
(springs,  dampers,  etc.). 

Controlled  Element  transfer  function. 

-  (lain  of  the  Controlled  Element. 

Laplace  operator. 

-  Intermittent  display  of  information.  The  picture 
painted  on  the  PPI  scope  during  one  sweep  decays  before 
the  next  sweep. 

The  relationship  between  an  Input  signal  and  the 
resulting  output  signal.  For  the  purpose  of  this 
study,  transfer  function  designates  the  relation¬ 
ship  of  a  movement  of  the  control  stick  to  the 
resulting  movement  of  the  oscilloscope  blip.  Figure 
♦  shows  the  relationship  of  the  movement  of  the 
control  handle  to  the  movement  of  the  oscilloscope 
blip  when  the  transfer  function  is  a  simple  gain, 

Y.  ■  10.  Figure  5  presents  the  resulting  response 
when  Yc  is  equal  to  £,  K  being  equal  to  1,  5,  and 
IS,  respectively. 


NAVTRADEVCEN  323-2 


0 


K  *  10 


LEGEND'  YC  =  K 

- INPUT  -  Control  •tick  position 

■— -1—-1  OUTPUT  -  P'ip  position 


-2  0  2  4  6  8  10 

TIME  (ssc) 


F/G4-  THE  RELATIONSHIP  OF  CONTROL  POSITION  TO  BUP  POSITION 
FOR  THE  FUNCTION  Ye  *  K 
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DISTANCE  FROM  CENTER  (arbitrary  unit) 
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TIME  (ft) 

neS.THt  RELATIONSHIP  OF  COHTnOL  Posmo*  70  BLIP  Posmo* 
APT  me  FUNCTION  Of  re  n-f 
(  Fore**  Function  ■  0 ) 
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